Skip to content

fix(agentic-engineering): forbid transferring a gh --json field name from a REST or GraphQL surface - #199

Merged
devantler merged 5 commits into
mainfrom
claude/surveyor-json-vocabulary-cross-surface-190
Sep 5, 2026
Merged

fix(agentic-engineering): forbid transferring a gh --json field name from a REST or GraphQL surface#199
devantler merged 5 commits into
mainfrom
claude/surveyor-json-vocabulary-cross-surface-190

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

The surveyor's rule against reusing gh --json field names only spoke of transfer between subcommands, yet the failures actually measured in a consuming deployment came overwhelmingly from a different API surface — a field name that is real in a REST payload being spent on a gh subcommand, which then rejects the entire read. One such read fanned across eight repositories turned a whole breakage sweep into "no evidence".

What

The rule now names the cross-surface case explicitly and calls out the two REST fields where the confusion originates, and the manifest validator pins the widened wording with its own failing fixture so it cannot regress silently.

Fixes #190

🤖 Generated with Claude Code

devantler and others added 3 commits September 5, 2026 18:05
…from a REST or GraphQL surface

The surveyor's vocabulary rule forbade transferring a `--json` field name between subcommands,
but 22 of 25 real `Unknown JSON field` failures measured in a consuming deployment came from a
different API surface: `path` learned from the REST `actions/runs` payload the prescribed
classifier consumes, then spent on `gh run list --json`, which rejects the whole read on one
unknown name — an entire default-branch breakage pass returning nothing for every repository.

Widen the rule to name the cross-surface case explicitly, call out `path` and `created_at` where
the definition points at that payload, and pin the widened wording in the manifest validator with
its own neutralising fixture (RED: nine pin rejections of the widened text under the old pin;
GREEN after). The plugin moves to 4.4.24 on top of the 4.4.23 base.

Fixes #190

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness record — tried and evaluated as a user at 5416b84

Definition prose with no runtime surface of its own; the validator that pins it is the exercisable user surface:

  • RED: with the widened bullet in the fixture stub and a new neutralising case in place, the unmodified validator rejected the widened wording nine times across the desired-state fixtures (174 passed, 8 failed, every failure carrying the vocabulary pin's error) — the old pin does not admit the new text.
  • GREEN: after widening the pinned string, 182 passed, 0 failed; the new case that rewrites the cross-surface clause to permit transfer fails as intended.
  • Real tree: ./scripts/validate-manifests.sh passes on the merged branch, refresh-desired-state-digests.sh --check is clean, and the version gate reports 4.4.23 → 4.4.24 on top of the base that fix(surveyor): require selection evidence before survey completion #193 moved.
  • Pin derivation: the pinned paragraph was generated from the surveyor text by whitespace normalisation and asserted to match before it was written, never retyped.

Consumer half: monorepo#3207 ported the subcommand-to-subcommand rule into the deployment overlay; this PR is the generic half #190 describes.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

@devantler I will review the changes in #199.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: 33e15982-f5c5-4361-be29-25abb1aa6bf8

📥 Commits

Reviewing files that changed from the base of the PR and between d5ee81a and 5416b84.

📒 Files selected for processing (8)
  • .claude-plugin/marketplace.json
  • .github/plugin/marketplace.json
  • plugins/agentic-engineering/.claude-plugin/plugin.json
  • plugins/agentic-engineering/agents/portfolio-surveyor.agent.md
  • plugins/agentic-engineering/plugin.json
  • plugins/agentic-engineering/resources/provider-neutral.desired-state.json
  • scripts/validate-manifests.sh
  • scripts/validate-manifests.test.sh

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

CodeRabbit refused this head on quota (Review rate limited), so the round advances to the next lane.

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T21:34:48.091656Z 5beb110 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 5416b84d5e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler
devantler marked this pull request as ready for review September 5, 2026 21:08
devantler and others added 2 commits September 5, 2026 23:11
…vocabulary-cross-surface-190

# Conflicts:
#	plugins/agentic-engineering/resources/provider-neutral.desired-state.json
…main now carries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

The merge with main (a sibling moved the plugin to 4.4.24) staled the earlier Codex green at 5416b84; this head carries the same change on top of the new base at 4.4.25.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== HEAD =='
git rev-parse HEAD
git status --short
git show --no-patch --format=fuller HEAD

printf '%s\n' '== Changed files against merge base with main =='
base="$(git merge-base HEAD origin/main)"
printf 'merge-base: %s\n' "$base"
git diff --name-status "$base"...HEAD
git diff --stat "$base"...HEAD

printf '%s\n' '== Diff =='
git diff --find-renames --find-copies --no-ext-diff "$base"...HEAD

printf '%s\n' '== Outlines of changed source-like files =='
git diff --name-only "$base"...HEAD | while IFS= read -r file; do
  case "$file" in
    *.js|*.mjs|*.cjs|*.ts|*.tsx|*.json|*.yaml|*.yml)
      printf '\n-- %s --\n' "$file"
      ast-grep outline "$file" --items all 2>/dev/null || true
      ;;
  esac
done

Length of output: 18680


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Validator contract and surrounding desired-state checks =='
sed -n '410,485p' scripts/validate-manifests.sh

printf '%s\n' '== Fixture setup, assertion helper, and changed tests =='
sed -n '620,710p' scripts/validate-manifests.test.sh
sed -n '915,985p' scripts/validate-manifests.test.sh
rg -n -C 3 'check_fail\(|make_desired_state\(|validate-manifests' scripts/validate-manifests.test.sh

printf '%s\n' '== Complete changed agent section =='
sed -n '1,110p' plugins/agentic-engineering/agents/portfolio-surveyor.agent.md

printf '%s\n' '== Agent definition digest verification =='
sha256sum plugins/agentic-engineering/agents/portfolio-surveyor.agent.md
python3 - <<'PY'
import json
with open("plugins/agentic-engineering/resources/provider-neutral.desired-state.json", encoding="utf-8") as f:
    data = json.load(f)
print(data["agents"]["portfolio-surveyor"]["definitionSha256"])
PY

printf '%s\n' '== Version declarations and contract references =='
rg -n -C 1 '"version": "4\.4\.(24|25)"|agentic-engineering.*4\.4\.(24|25)' \
  -g '*.json' -g '*.md' -g '*.sh' .
rg -n -C 2 'Every `gh --json` vocabulary is local to its subcommand|actions/runs|gh run list --json' \
  -g '*.md' -g '*.sh' -g '*.json' .

Length of output: 37108


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Desired-state JSON shape and SHA references =='
jq 'paths(scalars) as $p | select(($p | join(".")) | test("portfolio-surveyor|definitionSha256")) | {path: $p, value: getpath($p)}' \
  plugins/agentic-engineering/resources/provider-neutral.desired-state.json
rg -n -C 4 'portfolio_surveyor_sha256|actual_portfolio_surveyor_sha256|definitionSha256|portfolio-surveyor.*digest' \
  scripts/validate-manifests.sh scripts/validate-manifests.test.sh

printf '%s\n' '== Plugin version declarations =='
rg -n -C 1 '"version": "4\.4\.(24|25)"|agentic-engineering.*4\.4\.(24|25)' \
  -g '*.json' -g '*.md' -g '*.sh' . || true

printf '%s\n' '== JSON vocabulary contract references =='
rg -n -C 2 'Every `gh --json` vocabulary is local to its subcommand|actions/runs|gh run list --json' \
  -g '*.md' -g '*.sh' -g '*.json' . || true

printf '%s\n' '== Full validator assertion block =='
rg -n -A 90 -B 8 'portfolio_survey_json_vocabulary_contract' scripts/validate-manifests.sh

printf '%s\n' '== Test harness assertion helpers =='
sed -n '130,175p' scripts/validate-manifests.test.sh

Length of output: 40668


@devantler Review complete for head 5beb110861c33cb643dc524ed2956e172e42b01f.

I found no issues.

The surveyor rule, validator pin, negative fixture, version manifests, and desired-state SHA-256 are consistent. The fixture rejects the permitted cross-surface transfer as required.

You are interacting with an AI system.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness addendum at 5beb110 — merged with main after a sibling moved the plugin to 4.4.24, re-bumped to 4.4.25; generated digest conflict resolved by regeneration. validate-manifests.sh clean, self-test 185 passed, 0 failed, digests current, version gate 4.4.24 → 4.4.25, CI green at this head. Codex had already completed clean at the previous head 5416b84; the merge staled that green, so a fresh review is requested here.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

CodeRabbit answered the request as a chat prompt rather than a review (no review ran at this head), so the round advances to the next lane. Codex completed clean on this change at the previous head 5416b84; this head is the same change on top of the 4.4.24 base.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 5beb110861

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler
devantler merged commit 4cdc4bd into main Sep 5, 2026
41 checks passed
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 5, 2026
@devantler
devantler deleted the claude/surveyor-json-vocabulary-cross-surface-190 branch September 5, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

gh --json vocabulary rule covers subcommand-to-subcommand transfer, but 22 of 25 real failures come from REST/GraphQL surfaces

1 participant